Relevance & Research Question: While there is a major trend towards using big data from social media platforms, in particular Twitter data, to measure public opinion or even predict election/referendum outcomes, major questions remain about the representativeness and validity of such aggregate measures. How demographically and politically different are Twitter publics from the general public? How accurately can sentiment analysis capture actual opinions of individual Twitter users, and how can such opinions be meaningfully aggregated, given the asymmetry of participation on Twitter? We are using the UK Referendum on EU membership as a case study to trial an innovative approach towards validation and calibration of Twitter data.

Method & Data: After collecting 25.9 million of Referendum-related Tweets from a four month period prior to 23 June 2016, we fielded an online survey among Twitter users whose Tweets we have captured during this period. The 1,552 surveyed Twitter users and their 70,000 Tweets about half of which we expert-coded, provide us with a training data set. Finally, taking into account the variation in user activity on Twitter, we model our Twitter-derived proxy for public opinion as an aggregation of Tweets which are nested within individual users.

Results: Survey responses on EU attitude measures which correlate at over 0.8 with expert coding, serve both as a basis for supervised machine learning for sentiment analysis and for out-of-sample validation of sentiment scores. This provides us with a robust method of sentiment analysis to then apply to the full population of 25.9 million tweets.

Added Value: This method allows us not only to compare the socio-demographic and political composition of the Twitter public with the general UK public, but also to match a respondent’s Twitter output with their answers to our survey question. We can thus validate opinion measures from sentiment analysis by estimating how well these measures predict respondents’ attitudes towards EU membership derived from traditional survey questions. The survey then also allows for calibration, as we can compare sentiment distribution against the surveyed sample, as well as demographic distributions in the Twitter survey against known population distributions, to apply weights.